1. Field of the Invention
This invention relates to a method and apparatus for making and projecting three-dimensional digital images recorded on compatible media from optical images produced via the principles of holography and/or integral photography.
2. The Problem to be Solved and the Improved Solution
The 3-D PATENTS disclose and claim a SYSTEM AND APPARATUS FOR THE RECORDING AND PROJECTION OF IMAGES IN SUBSTANTIALLY 3-DIMENSIONAL FORMAT. The invention described therein derives from the principles of holography and/or integral photography. The 3-D PATENTS first disclose a basic principle of magnification and projection. This principle permits magnification and projection of three-dimensional images uniformly in all directions, thereby overcoming drawbacks in the prior art. My pending U.S. application Ser. No. 10/904,888 discloses a broad improvement upon this principle. Based upon this principle, cameras are described, in their various embodiments, that photograph a scene and retain the three-dimensional information therein. Although still life three-dimensional photographs made using this system can be simultaneously magnified and projected, the disclosed system has particular advantages in producing three-dimensional motion pictures. The 3-D PATENTS also describe an editor that would edit integral photographs and holograms containing the three-dimensional information from the photographed scene. In addition, a theater is designed wherein magnified three-dimensional photographed images are projected upon a large screen to be viewed by an audience. Further, the projectors and screens are described in their various embodiments. My pending U.S. application Ser. No. 10/416,689 discloses various screens to be used with this system. My pending U.S. application Ser. No. 10/904,745 discloses a unitary MODULAR INTEGRAL MAGNIFIER and a screen comprising these modules that is particularly useful for the Present Invention, and which is discussed later in the Present Application.
However, the system as described in the 3-D PATENTS emphasizes the use of photographic film as the recording medium. The preferred camera embodiment uses black-and-white film to record integral photographic images, and also uses color filters to retain the color information. This is done to compensate for the relatively low resolution of conventional color film. High resolution black-and-white film is commercially available. Higher film resolution means that more information can be recorded on and ultimately retrieved from the film medium. For example, ultra-high resolution black and white panchromatic film having a resolution of 2,000 line pairs per mm can be obtained. With this resolving power, information can be recorded on and retrieved from a spot as small as 0.25-μ (one-quarter micron). Yet, the information recording and retrieval requirements for a single frame or photograph of a three-dimensional picture using the system described in the 3-D PATENTS are so great as to exceed the resolution capabilities of even this film. Thus, the invention described therein proceeds to disclose a camera:                wherein a very large optical system is used to record horizontal parallax;        wherein vertical parallax is absent from the recorded picture so as to reduce the amount of required information;        wherein the recorded image is multiplexed on the film frame using a complex optical system so as to conserve space on the film;        wherein the film frame is larger than that of conventional motion picture cameras so as to increase the space available for recording the image;        wherein the film moves faster than in conventional motion picture cameras so as to increase the amount of information that can be recorded in a given time; and,        wherein the film is stabilized via a special three-point film motion registration mechanism.        
The 3-D PATENTS disclose a number of post processing methodologies wherein the integral photographic pictures remain as integral photographs or are converted to holograms. Ultimately, these photographs must be projected using a specially designed projector. The photographs to be projected reside either on photographic film or are stamped onto plastic using a conventional process. In either event, images from a moving physical medium (hereinafter “film”) are projected onto a special screen using the special projector. Depending upon the embodiment, the film is transported through a film motion mechanism either at constant velocity or in a manner similar to that used in conventional motion picture projectors wherein each frame of film is stopped for viewing. The film stabilization requirements make great demands upon the projector. One embodiment requires gyroscopic stabilization. Also, illumination demands are great, and very complex unmultiplexing optics are required for the projector.
Clearly, both the motion picture camera and the projector for the system disclosed in the 3-D PATENTS are much more expensive to fabricate than their current motion picture equipment counterparts. The fact that the camera is very expensive should not discourage its use since even its higher cost would represent a small fraction of the cost of currently producing a motion picture. Furthermore, a single camera can be used to produce a large number of motion pictures. However, display of the three-dimensional motion pictures so produced either requires that special theaters be built or that conventional theaters be converted to show these new films. History of the motion picture industry has demonstrated that special processes requiring special theaters to be built have not enjoyed enduring economic success. On the other hand, many special processes that have lent themselves to adaptation in conventional theaters have achieved lasting success. Conventional theaters that would show three-dimensional motion pictures produced using the described system would require a special screen and projector, and the audience seating would be limited to certain specified positions. A conversion design wherein there would be minimum disruption to current audience seating is feasible. One of the continuing patent applications to the 3-D PATENTS (No. 09/853,790) discloses a manufacturing process wherein the special screen of the preferred embodiment can be inexpensively fabricated as small rectangular tiles to be assembled as a larger screen of any desired dimensions in any theater. However, the projector is the most expensive component in a theater designed to display these three-dimensional motion pictures. The design element most responsible for the increased expense of this special projector relative to conventional motion picture projectors is the film motion mechanism and the stabilization mechanism resulting therefrom. Therefore, if it were possible to eliminate film from the projection process, projectors could be produced at a much lower cost than even conventional motion picture projectors. In fact, the Present Invention could entirely eliminate the need for a projector. Furthermore, as will be discussed later, two-dimensional motion pictures may be shown in the same theaters utilizing the same equipment.
Television is a current device that can display motion pictures optionally with or without the use of film. The final television display mechanism does not use film at all. Unfortunately, the resolution available from conventional television systems makes projection of motion pictures before large audiences in large theaters impractical. Computer monitors provide somewhat higher resolution than conventional television sets, and the new High Definition Television (HDTV) standard also addresses this problem. Home viewing audiences are gradually demanding larger television screens. Rear projection large screen video home entertainment systems are popular, but the video quality is much lower than with large screen conventional picture tubes. HDTV was created to expressly solve this problem. Occasionally, interactive technical seminars are held in large motion picture theaters wherein live and filmed two-dimensional video images are projected onto a large screen. While the quality of these pictures is acceptable to audiences attending such events, the quality of motion pictures so projected is very poor compared to the quality of conventional movies projected from film. Some attempts have been made to digitally project popular movies in a few theaters with limited success. The major drawback to currently available video projection systems is that the resolution is so poor compared to film projection that the audience must be seated far enough away from the screen so as not to notice the difference. The Digital Light Processor (DLP™) chip manufactured by Texas Instruments, Inc. provides improved resolution, but even when these chips are used in digital projectors, the resolution is far less than from conventional film projectors. In addition, the illumination systems of currently available video projectors produce less bright and lower contrast pictures than those produced by film projectors. This poor image quality would probably be acceptable to viewers were film projectors not to be the motion picture display standard. However, audiences always look for image quality improvement, and are unwilling to accept less than standard viewing conditions.
The three-dimensional motion pictures produced by the system described in the 3-D PATENTS cannot be transmitted over conventional broadcast television. The bandwidth is simply too low. A video frame, broadcast using the NTSC standard, possesses a total vertical resolution of 525 line pairs. The unit of video resolution is called the pixel. The number of pixels in a video frame is (320×525 resolvable spots) or 168,000 pixels. Some video frames are displayed using 640×480 pixels or 307,200 pixels. Compare this with a conventional motion picture frame on 35-mm color film where there are between five-million and twenty-million pixels depending upon the type of film used. The 3-D PATENTS place the three-dimensional motion picture frame resolution requirement between 400-million and 33.8-trillion pixels. A conventional television broadcast simply cannot transmit that much information at the rate of 30 frames per second to be able to produce the three-dimensional movies in this way. Other transmission methods, such as fiber optic cable, have sufficient bandwidth, but the NTSC, PAL, SECAM, and even the HDTV display standards do not provide enough monitor resolution to show these movies.
Videotape recording technology for television broadcasts has been around since the early 1960's. In the 1980's, home videocassette recorders became available. These video recording devices are analog in nature. This type of recording is the same as the television broadcast standard which is also analog. Recently however, cable providers have been broadcasting digital television programming, and digital video recorders are now available to the public. Digital video affords a noticeably better picture quality than analog video. DVD movies are gradually replacing videocassette movies in both home and professional entertainment systems. However, the NTSC, PAL, or SECAM standards used therein do not adequately take advantage of the capabilities of digital video. In the USA, the NTSC standard is expected to be phased out over the next few years in favor of the HDTV standard.
Even in the film industry, digital recording has become the standard. Very few motion pictures are still edited using mechanical editing machines. Instead, they are edited digitally, frame by frame on computers. Special effects are regularly produced by computers during post production, although some are still produced in camera. Frames of motion picture film are exposed or printed in high resolution from digital images stored in computers. However, for best quality, motion pictures must first be recorded on photographic film, then transferred to a digital medium for editing, and finally transferred back to film for display.
When a picture is recorded on photographic film, the picture resolution is defined by the product of the frame size (in square-millimeters) and the square of the film resolution (in line pairs per mm). Degradation of photographic resolution is referred to as graininess. When a picture is recorded digitally, it does not have a frame size. The picture is stored either in a computer or on a digitally compatible medium. The resolution of the picture is specified by its dimensions in pixels or by the total number of pixels it contains. Degradation in digital resolution is referred to as pixelation.
Digital recording of motion pictures is current state-of-the-art. NTSC, PAL, SECAM, or HDTV quality motion pictures can be captured using digital video cameras. Digital recordings are typically made on high quality videotape, CD-ROM, and DVD. They are also recorded and stored in computers. With many computers, the amount of information required to record an entire motion picture on the available storage media may be so large as to make such recording impractical. Therefore, software compression algorithms have been developed to reduce storage requirements. Much digital image compression is also state-of-the-art.
Still-life digital photographic images are currently stored by a computer in many formats. The common image file types are BMP (bitmapped), JPEG (Joint Photographic Experts Group), GIF (Graphics Interchange Format), JPEG 2000, TIFF (Tagged Image File Format), PNG (Portable Network Graphics), PSD (Photoshop), and PICT (developed for MacDraw software). BMP is an uncompressed file format that permits direct viewing of the pixels. The other image formats employ compression in one way or another that is either lossless or lossy (i.e., upon decompression, all of the image information may or may not be fully reconstructed). These image formats were developed specifically to enable computer images to be displayed on computer monitors. Typical monitor resolutions are 640×480, 800×600, and 1024×768 pixels. Therefore, the current image formats are inadequate to meet the resolution requirements of the three-dimensional process disclosed herein.
Storage of an uncompressed 24 bit per pixel digital video motion picture lasting two hours running at 29.97 frames per second (the NTSC standard) with a resolution of 640×480 would require approximately 1.6 terabytes. Sometimes motion pictures are actually stored as uncompressed digital video for editing purposes where computer graphics special effects are added to individual frames. One recent motion picture required 700 gigabytes of data storage for this purpose. In this case, the editors were unable to store the entire movie on their computer at one time. Individual scenes were downloaded from video tape, edited frame by frame, then re-recorded in edited form onto video tape, and deleted from computer memory to make room for the next scene. However, except for such demanding editing requirements, methods currently exist to enable entire video motion pictures to be practically stored on computers. In addition, the capacity of available storage media is increasing rapidly.
Motion picture digital video files are currently stored on computers using formats called codecs (enCOder/DECoder). These codecs are also used for distribution of motion pictures on the web or a LAN or any other method of file transfer. Most comprise both video and audio information and feature not only intra-frame compression but also inter-frame compression. AVI (Audio Video Interleave) is compressed with many different codecs depending upon the needs of the users. It is a special case of the RIFF (Resource Interchange File Format), and it is the most common format for audio/video data on the PC. Its specification was defined by Microsoft as was the ASF (Active Streaming Format). The MPEG (Motion Picture Experts Group) codecs are also extremely popular. The MPEG-1 format produces high quality video and audio streams at approximately 2×CD-ROM data rates. Its standard is full frame rate (24–30 fps, depending on the source) with a quarter size image (352×240), and it is useful for playback on most personal computers. The MPEG-2 format produces high data rate, full broadcast quality files. Its standard is full frame rate (24–30 fps) and full screen resolution (720×480). It is the format used for DVD-Video and many home satellite data systems. The standard for the Apple Macintosh computer is the Quicktime format. Many other formats, such as MPEG4 and others, are also popular. Unfortunately, these also are matched to current television sets or computer monitors, and are unsuitable for use with the three-dimensional motion picture format disclosed herein.
On the other hand, generalized data compression algorithms are not necessarily keyed to photographs or video files that require display on television sets or monitors. Instead, they perform their function on any computer data files based upon the redundancy of the information contained within those files. The following compression algorithms are popularly used:                Arithmetic Coding        Pulse Code Modulation        Differential Pulse Code Modulation        Run-Length Coding        Shannon-Fane Coding        Huffman Coding        Dictionary Methods        
The particular variant of arithmetic coding specified by the JPEG standard is the subject of patents owned by IBM, AT&T, and Mitsubishi. One of the most popular dictionary methods is LZW (Lempel-Ziv-Welch) compression. As an example, a TIFF image file can achieve almost 5:1 compression using this algorithm.
The ability to digitally store complete motion pictures and other types of television programming is recent. DVD players, which are digital, are now commonplace in the home, and they are beginning to replace older video cassette recorders (VCR's) which are analog. DVD players double as CD-ROM players in many desktop computers. AVI, MPEG-1, MPEG-2, MPEG-4, and Quicktime in addition to other codecs are commonly used to enable digital video programs to be stored on the hard drives of desktop computers. Using appropriate software, computer users can watch these movies and programs on their computer monitors, and, in some cases, on their television sets.
Many companies have combined this technology with that of the internet. It would be desirable to transmit digital motion pictures and other video programming from computer to computer. Unfortunately, they were confronted with bandwidth problems. This problem can be illustrated by examining the available bandwidth for several methods of data delivery:
TECHNOLOGYDATA TRANSFER RATEFast Ethernet100MbpsEthernet10MbpsCable Modem8MbpsASDL6Mbps1x CD-ROM1.2MbpsDual channel ISDN128KbpsSingle channel ISDN64KbpsHigh speed modem56KbpsStandard modem28.8Kbps
Uncompressed video at 216 Mbps and above cannot be transmitted using these methods of data delivery. Using a cable modem and depending upon internet traffic, it can sometimes take as much as twenty-four hours to practically download a complete motion picture stored in AVI format. An additional problem arises when a user downloads a program from a host computer. By doing this, the user now has a perfect copy of the original program, and he can potentially re-distribute the program to others without the knowledge or permission of the program's owner. To overcome these problems, video streaming technology was developed. With video streaming, files can play as they are being downloaded to the user's computer. Thus the necessity for downloading and storing entire files before playback has been eliminated. In the video streaming model, information from the digital video file is sent by the video server to a data buffer. Data packets are then sent across communication lines to the client computer where it is stored in a data buffer and is played as it is received. These files are usually transmitted using AVI, ASF, H.263, or MPEG-4 codecs as well as proprietary streaming codecs such as Vxtreme, ClearVideo, VDOLive, Vivo, RealVideo, TrueStream, and Xing. Video streaming has specific hardware and software requirements. Systems combining both are commercially available. Generic video streaming software is also available.
The problems with currently available video streaming technology are the dependence upon relatively low available bandwidths and the incapability of producing programming with acceptable resolution on commercially available computer monitors. Even were these problems to be solved for conventional video programming, it would not be even nearly adequate for the much higher data transfer rates required for three-dimensional motion pictures produced using the process disclosed herein.
It has already been discussed that elimination of the film requirements of the invention in the 3-D PATENTS would be desirable for reasons of cost reduction. Digital projection and/or transmission provides the solution to this problem and is therefore an object of this invention. The present invention, in all its embodiments, is realized by a camera that digitally records the integral photographic images and by a mechanism that magnifies these digital images before a large audience. The principle for magnifying these digital images is identical to that claimed in the 3-D PATENTS, and described therein as “the magnification principle.” However, digital images can also be created by direct recording or by transfer from photographic images on film. Therefore, this system is compatible with photographic recording using cameras disclosed in the 3-D PATENTS.
As the 3-D PATENTS' disclosure proceeded, it became apparent that, once the basic principle of magnification and projection was utilized, the primary technological problem centered around practically recording a sufficient quantity of information upon a manageable amount of photographic film. Much of the disclosures of the 3-D PATENTS describe the various solutions to this problem. In the Present Invention, it will become apparent that, once “the principle of magnification” is employed, the primary technological problem centers around practically recording a sufficient quantity of information on digital recording devices as well as on digitally compatible recording media. Much of the disclosure for the Present Invention addresses this problem. Another major technological problem, the solution to which is addressed in the Present Invention, is integration and transmission of the vast quantity of digital information using digital electronics so as to photograph large three-dimensional scenes and to project or transmit the three-dimensional pictures produced therefrom onto a screen for viewing.
The 3-D PATENTS disclosed that the system described therein can be used with television systems, computers, video recording and animation. They disclosed that it is possible to construct a home entertainment system (resembling a television set) that employs the methods and apparatus described therein. Furthermore, while broadcasting of three-dimensional photographic material over conventional television broadcast bands or even over conventional cable TV transmission is unfeasible due to bandwidth considerations, it is possible to adequately transmit such image information over closed circuit fiber optics cable. This has implications not only for home entertainment but also for computer displays. Using the methods and apparatus described therein, three dimensional computer graphics is feasible. Furthermore, the use of magnetic videotape (or other magnetic media) to record the information necessary for image retrieval using this method is also feasible. So also is the use of CD's of various formats (e.g., laser disc and DVD) feasible for this purpose.
In view of the above, it is therefore an object of the present invention to provide a three-dimensional system and method in which non-stereoscopic images can be magnified and digitally projected or transmitted to be seen by large audiences. Digital cameras and digital projectors will be described to accomplish this objective. Digital projectors are not necessary for all embodiments. Editing will be accomplished digitally. Another object of the invention is to provide such a system wherein said images are still life and/or moving pictures. Yet another object of the invention is to provide a three-dimensional system which is adaptable for use in animation, home entertainment, and computer technology. A further object of the invention is to enable the invention described herein to be compatible with display of two-dimensional images.
Factors Distinguishing the Present Invention Over the Prior Art
The concept of integral photography is not new. Edward Muybridge was the first to produce multiple photographs of a given scene. He obtained a patent in 1883. His process used multiple cameras to photograph moving objects. Each camera photographed the same scene from a slightly different view point. His now famous photographs were later assembled as frames to be used in the earliest motion pictures. The distinguishing feature of his process is that the photographs were not exposed simultaneously, but rather in a temporally sequential sequence. A few years later, Henry Kuhn developed a process to photograph a single scene from multiple viewpoints apparently simultaneously. The elemental photographs so produced were assembled onto a single photographic sheet arranged as a “stamp portrait.” In 1920, J. W. Legg perfected a high speed camera that used multiple exposures to produce a series of stereoscopic photographs. In the meantime, several cameras were developed to produce photographic stereoscopic pairs. These stereoscopic pairs could be later reconstructed as three-dimensional scenes in special viewers called stereoscopes. At the time, stereoscopes had been used for at least a century to view artistic drawings in three-dimensions. Stereoscopes are still sold today. Stereoscopic cameras represented an improvement that enabled people to view photographed scenes with three-dimensional realism. In 1930, Herbert Ives developed a process to produce “parallax panoramagrams” from a pair of stereoscopic photographs. “Parallax panoramagrams” used lenticular sheets to reconstruct three-dimensional scenes without requiring additional special viewing aids. Ernest Draper, in 1934, developed a method of using multiple cameras to simultaneously photograph a three-dimensional scene. In 1936, Douglas Coffey developed a process for producing three-dimensional lenticular integral photographs called “composite stereographs.” During the ensuing years, many improvements on these processes were made, and lenticular stereograms and integral photographs are still currently made and sold.
Until the invention of the system and apparatus described by the applicant in the 3-D PATENTS, integral photographs could not be practically magnified nor projected before a large viewing audience. Projection of non-stereoscopic three-dimensional motion pictures in theaters was heretofore unfeasible. In the 3-D PATENTS, integral photographs are produced using a first optical system comprised of a matrix lens array and other optical elements. The matrix lens array consists of a plurality of elemental lenses. Each elemental lens is capable of producing a single two-dimensional elemental photograph. The combination of these elemental photographs forms a two-dimensional integral photograph. The integral photograph is then magnified by a given magnification factor. The magnification process produces either a new integral photograph or a projected image. When the magnified two-dimensional integral photograph is viewed through a second optical system, a correctly magnified three-dimensional scene can be viewed. The second optical system is similar to the first in that it was scaled up from the first optical system by the magnification factor.
The same principle of magnification and projection described in the 3-D PATENTS, is used in the present invention. The difference is that video and digital imaging is used instead of photographic film. Consequently, the integral photographs of the 3-D PATENTS, do not physically exist in the present invention. Instead, they are virtual digitizations initially created with video imaging tubes and subsequently stored as digital information in computer memory and peripheral media. Virtual integral photographs produced in this manner are novel. The actual scenes are photographed using camera embodiments described herein. Many of the camera optical systems resemble those of the cameras described in the 3-D PATENTS. However, as will be described herein, because of image resolution problems, multiple video imaging tubes must be used. In most of the camera embodiments described, a single integrated first active optical system is used with a plurality of imaging devices. Yet, in one of the embodiments, a plurality of conventional video cameras functioning together as a single unit are used to produce the digitized virtual integral photograph. This camera embodiment is reminiscent of the earlier prior art discussed above. This is only an external physical resemblance due to the use of multiple cameras in the inventions dating back to the nineteenth century. However, when considered as a unit, this camera embodiment is not at all anticipated nor predicted by the aforementioned prior art. An integral photograph is composed of a plurality of elemental photographs, each elemental photograph requiring a separate lens for its creation. Whether that lens belongs to an integrated optical system or is a component of a separate single camera is irrelevant. A single camera with multiple lenses and imaging devices is the equivalent of the same number of multiple cameras having a single lens and single imaging tube. The unique aspect of this camera embodiment as well as of the alternate camera embodiments is the combination of camera components designed to produce the appropriate digitized virtual integral photograph. In the preferred camera embodiments, not all of the required elemental photographs are produced. The number of elemental photographs so created is insufficient to combine to produce an adequately viewable three-dimensional scene. To solve this problem, the preferred camera embodiments also contain a computer module to derive the missing elemental photographs by interpolation.